Information processing apparatus and method for registering moving objects as metadata of an image

ABSTRACT

An association degree evaluation unit acquires pieces of position information of an image sensing apparatus at respective times within an adjacent time range to an imaging time of a designated image of those sensed by the image sensing apparatus. Furthermore, the association degree evaluation unit acquires pieces of position information of a moving object at the respective times within the adjacent time range. Then, the association degree evaluation unit calculates a similarity between routes of the image sensing apparatus and moving object based on the acquired position information group, and decides a degree of association between the designated image and moving object based on the calculated similarity. An associating unit registers information indicating the degree of association in association with the designated image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for associating an imagewith information related to this image.

2. Description of the Related Art

Conventionally, a method of automatically assigning information such asa person related to an image by recognition processing from a face imageof an object(http://www.sony.co.jp/SonyInfo/technology/technology/theme/sface_(—)01.html(updated Jan. 13, 2009)), and a method of making an imaging target holda digital tag, reading the digital tag by a camera, and automaticallyassociating an image and the target (Japanese Patent Laid-Open No.2005-086759) have been proposed. Also, an apparatus, which acquiresposition information at the time of imaging using a GPS equipped in acamera, and assigning that information as metadata of a sensed image hasbeen proposed (Japanese Patent Laid-Open No. 2001-292411).

However, the conventional method of identifying a face image is premisedon that a face appears in an image. Even when a face exists at thatsite, if it does not appear in an image, no information can beassociated with the image. Alternatively, since a side-faced object orhis or her back shot cannot be identified, it is difficult to associateinformation with that image.

When a digital tag or the like is used, a reader for reading informationof the digital tag has to be added to a camera. In this case, there arelimitations and problems that an imaging target has to hold, in advance,a digital tag or the like which is to be read by the reader, and animage of that target has to be sensed while the reader faces the target.

Also, position information at the time of imaging can be assigned to animage as metadata. However, it is impossible to assign information aboutwho joins an event including imaging as metadata.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theaforementioned problem and provides a technique for associatinginformation related to an image with this image by a simpler method.Also, the present invention improves the precision of recognitionprocessing of an object as information related to an image when thatprecision is insufficient.

According to the first aspect of the present invention, an informationprocessing apparatus comprising:

a first acquisition unit which acquires a plurality of pieces ofposition information of an image sensing apparatus at times adjacent toan imaging time of an image by the image sensing apparatus;

a second acquisition unit which acquires a plurality of pieces ofposition information of a moving object at the adjacent times;

a calculation unit which calculates a similarity between a route of theimage sensing apparatus and a route of the moving object based on theplurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object; and

a registration unit which registers as metadata of the image, when thesimilarity is higher than a pre-set threshold, information used tospecify the moving object.

According to the second aspect of the present invention, an informationprocessing method comprising:

a first acquisition step of acquiring a plurality of pieces of positioninformation of an image sensing apparatus at times adjacent to animaging time of an image by the image sensing apparatus;

a second acquisition step of acquiring a plurality of pieces of positioninformation of a moving object at the adjacent times;

a calculation step of calculating a similarity between a route of theimage sensing apparatus and a route of the moving object based on theplurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object; and

a registration step of registering as metadata of the image, when thesimilarity is higher than a pre-set threshold, information used tospecify the moving object.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the hardware arrangementof a computer 100;

FIG. 2 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to thefirst embodiment;

FIG. 3 is a flowchart of processing to be executed by the computer 100;

FIG. 4 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to thethird embodiment;

FIG. 5 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to thefifth embodiment;

FIG. 6 shows an example of the configuration of a management table;

FIG. 7 is a view for explaining routes of an image sensing apparatus andmoving objects on a two-dimensional plane;

FIG. 8 is a table showing distances between the image sensing apparatusand moving objects A, B, and C;

FIG. 9 is a view for explaining routes of an image sensing apparatus andmoving objects on a two-dimensional plane;

FIG. 10 is a table showing distances calculated by an association degreeevaluation unit 204;

FIG. 11 is a table showing scores;

FIG. 12 is a table showing the correction result of a score of object Aby an object recognition unit 501;

FIG. 13 is a table showing an example of the configuration of routeinformation;

FIG. 14 is a view showing a display example in the fourth embodiment;

FIG. 15 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to theseventh embodiment; and

FIG. 16 is a flowchart of processing to be executed by the informationprocessing apparatus according to the seventh embodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described hereinafter withreference to the accompanying drawings. Note that each of theembodiments to be described hereinafter is an example when the presentinvention is practiced, and is one of practical embodiments of thearrangement described in the scope of the claims.

First Embodiment

FIG. 1 is a block diagram showing an example of the hardware arrangementof a computer 100 applicable to an information processing apparatusaccording to this embodiment. Note that the arrangement shown in FIG. 1is implemented by one computer 100 in this embodiment. However, anarrangement equivalent to that shown in FIG. 1 may be implemented by aplurality of computers. When this embodiment is implemented by aplurality of computers, these computers make data communications witheach other via a network such as a LAN.

Referring to FIG. 1, a CPU 101 controls the overall computer 100 usingcomputer programs and data stored in a ROM 102 and RAM 103, and executesrespective processes to be described later as those which are to beimplemented by the computer 100.

The ROM 102 stores setting data, a boot program, and the like of thecomputer 100. The RAM 103 has an area used to temporarily store computerprograms and data loaded from an external storage device 104, and thoseexternally received via a network interface 107. Furthermore, the RAM103 has a work area used when the CPU 101 executes various processes.That is, the RAM 103 can provide various areas as needed.

The external storage device 104 is a large-capacity storage devicerepresented by a hard disk drive. The external storage device 104 savesan OS (Operating System), and computer programs and data required tomake the CPU 101 execute various processes to be described later asthose which are to be implemented by the computer 100. Also, theexternal storage device 104 saves information described as giveninformation in the following description. The computer programs and datasaved in the external storage device 104 are loaded as needed onto theRAM 103 under the control of the CPU 101, and are to be processed by theCPU 101.

Note that other external storage devices of the computer 100 may be usedin addition to the external storage device 104. For example, a memorycard, flexible disk, optical disk such as a Compact Disk (CD), magneticand optical cards, IC card, and the like are applicable.

An input device 109 includes a keyboard and mouse. When the user of thecomputer 100 operates the input device 109, he or she can input variousinstructions to the CPU 101.

An input device interface 105 is used to connect the input device 109 tothe computer 100. The CPU 101 is notified of an operation instruction,which is input when the user operates the input device 109, via thisinput device interface 105 and a system bus 108.

An output device 110 is represented by a display device including a CRTor liquid crystal display, and can output data in an arbitrary form.Data to be output which is processed in the computer 100 is output tothis output device 110 via an output device interface 106.

The network interface 107 is used to connect the computer 100 to anetwork line 111 such as a LAN or the Internet. The computer 100 canmake data communications with apparatuses on the network line 111 viathe network interface 107.

FIG. 2 is a block diagram showing an example of the functionalarrangement of the information processing apparatus according to thisembodiment. As shown in FIG. 2, the information processing apparatusaccording to this embodiment includes an image metadata management unit201, image sensing apparatus route management unit 202, associationdegree evaluation unit 204, associating unit 205, and moving objectroute management unit 203. Note that the respective units shown in FIG.2 are saved in the external storage device 104 as computer programs inthis embodiment. Alternatively, these units may be implemented byhardware.

The image metadata management unit 201 manages the following pieces ofinformation for each image sensed by the image sensing apparatus. Thatis, the unit 201 manages, as a set for each image, pieces of auxiliaryinformation (metadata) including imaging position information indicatinga position (imaging position) of the image sensing apparatus at the timeof imaging, imaging time information indicating an imaging time, andapparatus information used to specify this image sensing apparatus. Suchset is acquired, for example, as follows.

Assume that one image is sensed using the image sensing apparatus. Atthis time, imaging position information obtained from a GPS (GlobalPositioning System) equipped in the image sensing apparatus, imagingtime information measured by a timer incorporated in the image sensingapparatus, and apparatus information such as an ID unique to the imagesensing apparatus are acquired from this image sensing apparatus. Thesepieces of acquired information are registered as metadata in theexternal storage device 104 together with the sensed image. Theserespective pieces of registered information are managed by the imagemetadata management unit 201.

Note that the metadata may be registered in the external storage device104 in various modes. For example, the metadata may be embedded in thesensed image in a specific format, thereby registering the metadata inthe external storage device 104 together with the image. Alternatively,the metadata may be registered in the external storage device 104 or anexternal database as a file independent of the image. That is, theregistration mode is not particularly limited as long as image data andmetadata can be handled in association with each other. In the followingdescription of this embodiment, assume that an image body and metadataare registered in the external storage device 104 as independent files,and pieces of information associated with them are managed by the imagemetadata management unit 201.

FIG. 6 shows an example of the configuration of a management table usedto manage an image file registered in the external storage device 104and metadata for this image file. This management table is managed bythe image metadata management unit 201.

An ID (image ID) unique to a sensed image is registered in a field 610.When an image is registered as a file in the external storage device104, a path name of this image file in the external storage device 104is registered in a field 620. Apparatus information as informationunique to an image sensing apparatus which was used to sense an image isregistered in a field 601. Imaging time information of this image isregistered in a field 602. Imaging position information of this image isregistered in a field 603.

An example shown in FIG. 6 indicates that an image with an image ID“XXXX” is registered in the external storage device 104 to have a path“c:¥photo”. Furthermore, an image sensing apparatus used to sense thisimage is “Cano Shot G9”, its imaging time is “2008/9/26 12:00 AM”, andits imaging position is “N35.56564 E139.68129”.

In this way, every time the image sensing apparatus senses an image,pieces of information described above corresponding to the fields 610,620, 601, 602, and 603 are registered for each sensed image. The imagemetadata management unit 201 executes this registration processing. Thesensed image may be either a still image or movie. In this embodiment,assume that each sensed image is a still image.

The image sensing apparatus route management unit 202 manages, for eachimage sensing apparatus, route information that records pieces ofposition information of the image sensing apparatus (image sensingapparatus position information) measured at respective predeterminedtimings (periodically) (first management). The acquisition mode of theimage sensing apparatus position information is not particularlylimited. For example, as described above, pieces of image sensingapparatus position information are periodically acquired from the GPS(Global Positioning System) equipped in the image sensing apparatus, andrespective pieces of acquired image sensing apparatus positioninformation may be registered in the route information. Note that theroute information for each image sensing apparatus is associated withimage sensing apparatus information of the corresponding image sensingapparatus.

The moving object route management unit 203 manages, for each movingobject, route information that records pieces of position information ofa moving object such as a person or vehicle for respective times (movingobject position information) (second management). The acquisition modeof this route information is not particularly limited. For example, whena moving object is a person, pieces of position information forrespective times of the person may be recorded using a GPS function ofhis or her mobile phone, and may be acquired and managed by the movingobject route management unit 203 as route information of that person.Alternatively, when one is in automobile, history information recordedby a car navigation system may be acquired as route information, andthat information may be managed by the moving object route managementunit 203. Note that the route information for each moving object isassociated with information (moving object information) unique to thecorresponding moving object.

Note that the route information managed by each of the image sensingapparatus route management unit 202 and moving object route managementunit 203 is registered with pieces of position information at respectivetimes, and has a configuration, as shown in, for example, FIG. 13. FIG.13 shows an example of the configuration of the route information. InFIG. 13, pieces of position information (latitude, longitude) areregistered at 5-sec intervals. Note that the route information managedby the image sensing apparatus route management unit 202 and movingobject route management unit 203 may be uploaded onto an externalapparatus depending on its data size.

Referring back to FIG. 2, the association degree evaluation unit 204calculates a degree of association between a designated image anddesignated moving object using pieces of information respectivelymanaged by the image metadata management unit 201, image sensingapparatus route management unit 202, and moving object route managementunit 203. Processing for calculating the degree of association will bedescribed later.

The associating unit 205 decides based on the degree of associationcalculated by the association degree evaluation unit 204 whether or notthe designated image and designated moving object are to be associatedwith each other. The associating unit 205 registers informationindicating the degree of association in the external storage device 104as metadata for the designated image.

FIG. 3 is a flowchart of processing to be executed by the computer 100so as to associate an image and moving object with each other. Note thatthe respective units shown in FIG. 2 will be explained as main bodies ofthe processing below. However, as described above, since all of theseunits are implemented by computer programs, the CPU 101 which executesthese computer programs serves as a main body of the processing inpractice.

The user makes an operation for designating an image and moving objectto be associated with each other using the input device 109. Forexample, a list of an image group and pieces of moving objectinformation of respective moving objects, which have already beenregistered in the external storage device 104, is displayed on a displayscreen of a display device as the output device 110. The user designatesone image and one moving object information using the input device 109.Therefore, the association degree evaluation unit 204 receives anoperation instruction from the user in step S301. This operationinstruction includes an ID of the image and moving object informationdesignated by the user.

Of course, the designation method of an image and moving object astargets in the following processing is not particularly limited. Forexample, an image and moving object may be input using a userinstruction outside the computer 100 or may be designated by internalprocessing in the computer 100. For example, when the CPU 101 detectsthat a new image is registered in the external storage device 104, itmay internally and automatically select a moving object in turn frommoving object route information which is close to the imaging time andimaging position of this image.

In step S302, the association degree evaluation unit 204 acquiresimaging time information, imaging position information, and apparatusinformation, which are managed by the image metadata management unit 201in association with the ID of the image included in the operationinstruction received in step S301.

In step S303, the association degree evaluation unit 204 executes thefollowing processing (first acquisition). That is, the unit 204 acquirespieces of image sensing apparatus position information at respectivetimes within a time range adjacent to an imaging time indicated by theimaging time information acquired in step S302 from route informationwhich is managed by the image sensing apparatus route management unit202 in association with the apparatus information acquired in step S302.Note that in this step the unit 204 may acquire the route informationitself which is managed by the image sensing apparatus route managementunit 202 in association with the apparatus information acquired in stepS302.

In step S304, the association degree evaluation unit 204 executes thefollowing processing (second acquisition). That is, the unit 204acquires pieces of moving object position information at respectivetimes within the adjacent time range from route information which ismanaged by the moving object route management unit 203 in associationwith the moving object information included in the operation instructionacquired in step S301. Note that in this step the unit 204 may acquirethe route information itself, which is managed by the moving objectroute management unit 203 in association with the moving objectinformation included in the operation instruction acquired in step S301.

In step S305, the association degree evaluation unit 204 compares theimage sensing apparatus position information group (partial routeinformation of the image sensing apparatus) acquired in step S303 andthe moving object position information group (partial route informationof the moving object) acquired in step S304. With this comparison, theunit 204 calculates a similarity between the route of the image sensingapparatus and that of the moving object within the adjacent time range.Various methods of calculating the similarity of the routes areavailable, and an arbitrary method may be used. In this embodiment, theunit 204 evaluates, using this similarity, a degree of associationbetween the image and moving object, which are designated by the user.

An example of processing for evaluating the degree of associationbetween the image and moving object, which are designated by the user,will be described below with reference to FIGS. 7 and 8. FIGS. 7 and 8will explain a case in which the degree of association between each of aplurality of moving objects and an image is to be evaluated. However,this explanation does not limit the number of moving objects.

FIG. 7 is a view for explaining routes of the image sensing apparatusand moving objects on a two-dimensional plane. Referring to FIG. 7,reference numeral 701 denotes a route of the image sensing apparatus.Reference numeral 702 denotes a route of moving object A; 703, that ofmoving object B; and 704, that of moving object C. Assume that theseroutes 701 to 704 are those within a certain time range (2008/9/26 9:00to 15:00). Reference numeral 705 denotes an imaging position at animaging time “2008/9/26 12:00” of the image designated by the user.Also, “s” on each route indicates the position on that route at time Ts30 minutes before the imaging time “2008/9/26 12:00”. Furthermore, “e”on each route indicates the position on that route at time Te 30 minutesafter the imaging time “2008/9/26 12:00”.

The association degree evaluation unit 204 executes the followingprocessing. That is, the unit 204 calculates, using the respectivepieces of route information, distances between the image sensingapparatus and moving objects A, B, and C at respective times (forexample, divided times when a time range (in this case, a Ts-to-Terange) including the imaging time of the designated image is divided atequal intervals) within that time range. FIG. 8 is a table showing thedistances between the image sensing apparatus and moving objects A, B,and C at respective times within the Ts-to-Te range. In FIG. 8,“2008/9/26 12:00” is indicated by Tn.

The association degree evaluation unit 204 calculates a distance betweenthe position of the image sensing apparatus and that of moving object Aat time Ts using the image sensing apparatus position information groupacquired in step S303 and moving object position information group ofmoving object A acquired in step S304. In FIG. 8, this distance is“1.3”. Likewise, the association degree evaluation unit 204 calculates adistance between the position of the image sensing apparatus and that ofmoving object B at time Ts using the image sensing apparatus positioninformation group acquired in step S303 and moving object positioninformation group of moving object B acquired in step S304. In FIG. 8,this distance is “5.5”. Likewise, the association degree evaluation unit204 calculates a distance between the position of the image sensingapparatus and that of moving object C at time Ts using the image sensingapparatus position information group acquired in step S303 and movingobject position information group of moving object C acquired in stepS304. In FIG. 8, this distance is “10.0”.

In this way, the association degree evaluation unit 204 calculates thedistances between the image sensing apparatus and moving objects A, B,and C at respective times within the range from time Ts to time Te. Notethat the method of calculating a distance is not particularly limited,and various methods may be applied. For example, differences between thelatitude and longitude values may be used intact as a distance, or theymay be converted into a distance (meters).

Next, the association degree evaluation unit 204 calculates a variancevalue and average value of the distances calculated for moving object A,those of the distances calculated for moving object B, and those of thedistances calculated for moving object C. Since it is considered that amoving object with the smaller calculated variance value and averagevalue has a higher similarity between the route of the apparatus used tosense the image designated by the user and that of this moving object,the unit 204 evaluates that a degree of association between these imageand moving object is high.

According to FIG. 8, the distances between the positions of the imagesensing apparatus and those of moving object A at respective times aresmall, and their average and variation (variance value) are also small.In such case, the association degree evaluation unit 204 determines thata similarity between the route 701 of the image sensing apparatus andthe route 702 of moving object A is high, and evaluates that a degree ofassociation between the image sensed at the imaging time “2008/9/2612:00” and moving object A is high.

On the other hand, as for moving object B, the distance from the imagesensing apparatus at time Tn is relatively small, but the distances attimes Ts and Te are relatively large. Although the average value of thedistances between the positions of the image sensing apparatus and thoseof moving object B at respective times is relatively small, theirvariance value is larger than that of moving object A. For this reason,the association degree evaluation unit 204 evaluates that moving objectB has a lower degree of association with the image sensed at the imagingtime “2008/9/26 12:00” than moving object A.

On the other hand, the shape of the route 704 of moving object C issimilar to that of the route 701 of moving object A. However, thepositions of moving object C at respective times within the range fromtime Ts to time Te are relatively largely different from those of theimage sensing apparatus at respective times within the range from timeTs to time Te. For this reason, according to FIG. 8, the distancesbetween the positions of the image sensing apparatus and those of movingobject C at respective times are large, and their average and variation(variance value) are also large. In such case, the association degreeevaluation unit 204 determines that a similarity between the route 701of the image sensing apparatus and the route 704 of moving object C islower than moving object A. Therefore, the unit 204 evaluates that adegree of association between the image sensed at the imaging time“2008/9/26 12:00” and moving object C is lower than moving object A.

In this embodiment, the degrees of association are evaluated using thesimilarities of the routes, as described above. Note that thesimilarities have been described using abstract expressions (forexample, a similarity is high/low). However, in practice, a threshold isset for numerical values to make stricter judgments.

For example, the association degree evaluation unit 204 evaluates adegree of association between the image and moving object selected bythe user using a calculated average value X and variance value (standarddeviation) Y. The association degree evaluation unit 204 determines asimilarity between the routes using the average value X and variancevalue Y. When the average value X meets X≦th1 with respect to athreshold th1, and the variance value Y meets Y≦th2 with respect to athreshold th2, the unit 204 judges that a similarity is high. Otherwise,the unit 204 judges that a similarity is low.

When the association degree evaluation unit 204 judges that thesimilarity is high, it evaluates that a degree of association betweenthe image and moving object designated by the user is “high”. When theunit 204 judges that the similarity is low, it evaluates that a degreeof association is “low” (no association).

In this way, the degree of association between the image and movingobject is evaluated based on the average value X and variance value Y.Note that the number of evaluation stages is not limited to two. Forexample, three evaluation stages may be used as follows.

Upon calculating the average value X, a weighted average value iscalculated by giving larger weights as the moving object is temporallycloser to the designated image. When the weighted average value X meetsX≦th1 with respect to the threshold th1, and the variance value Y meetsY≦th2 with respect to the threshold th2, the association degreeevaluation unit 204 judges that a similarity is “high in the entireroute”. At this time, the unit 204 evaluates a degree of associationlike “acted together”. When the average value X meets X≦th1 with respectto the threshold th1, and the variance value Y meets Y>th2 with respectto the threshold th2, the unit 204 judges that a similarity is “highnear the selected image”. At this time, the unit 204 evaluates a degreeof association like “stayed in close at the time of imaging of theselected image”. In other cases, the unit 204 judges that a similarityis “low”, and evaluates a degree of association as “no association”.

Referring back to FIG. 3, in step S306 the associating unit 205associates the image and moving object which are designated by the userin accordance with the evaluated degree of association based on thedegree of association evaluated in step S305.

In step S307, the associating unit 205 registers information indicatingthe contents associated in step S306 in the external storage device 104as new metadata of the image designated by the user. For example, whenthe association degree evaluation unit 204 evaluates the degree ofassociation as “acted together” in the aforementioned example, the unit205 registers information indicating a moving object acted together asthat indicating this evaluation. For example, a name of a member whotraveled together is registered in metadata of the image.

As described above, according to this embodiment, a degree ofassociation between a moving object such as a person and an image iscalculated based on the imaging time information and imaging positioninformation of the image and a similarity between the route informationof the image sensing apparatus used to sense this image and that of themoving object, and they are associated with each other according to thedegree of association. In this way, information of, for example, aperson related to each individual image can be efficiently and preciselyassigned independently of whether or not that person appears in theimage.

Second Embodiment

In the first embodiment, the image sensing apparatus route managementunit 202 acquires route information of an image sensing apparatus fromthat image sensing apparatus. However, the image sensing apparatus neednot always hold the route information of the image sensing apparatus.

For example, “when” the image sensing apparatus was ready to communicatewith “which” GPS terminal such as a mobile phone is recorded using anear field wireless communication technique such as UWB (UltraWideband). Then, based on that record, pieces of route information ofrespective GPS terminals which were located near the image sensingapparatus are joined to acquire route information of the image sensingapparatus.

Alternatively, a photographer who held the image sensing apparatus isspecified, and route information may be acquired from a GPS terminalsuch as a mobile phone of the specified photographer. However, since theimage sensing apparatus may be shared by a plurality of photographers, amechanism that allows identifying a photographer who holds the imagesensing apparatus accordingly is required. For example, a photographeris identified using biometrics such as fingerprint authentication torecord “when” and “who” held the image sensing apparatus. Thus, piecesof route information while each photographer held the image sensingapparatus are joined to find the route of the image sensing apparatus.

According to these methods, the route information of the image sensingapparatus can be acquired without providing any arrangement foracquiring position information to the image sensing apparatus. Notethat, for example, an owner of the image sensing apparatus is oftendecided to have a role as a photographer. In such case, the routeinformation of a GPS terminal of the photographer may be used intact asthat of the image sensing apparatus. This method may have insufficientprecision as route information of the image sensing apparatus comparedto the route information measured by the image sensing apparatus or themethod using communications between the GPS terminal and image sensingapparatus. However, this method has an advantage of easily acquiringroute information without adding any arrangement to the image sensingapparatus.

Third Embodiment

In the first embodiment, the image sensing apparatus route managementunit 202 manages route information of an image sensing apparatus, and adegree of association between an image and moving object is decided bycomparing the managed route information and that managed by the movingobject route management unit 203. This embodiment will explain atechnique that can achieve the same object without using the routeinformation of the image sensing apparatus. Note that only differencesfrom the first embodiment will be explained below.

The hardware arrangement of an information processing apparatusaccording to this embodiment uses that shown in FIG. 1, and thefunctional arrangement adopts that shown in FIG. 4. FIG. 4 is a blockdiagram showing an example of the functional arrangement of theinformation processing apparatus according to this embodiment. Thisembodiment uses imaging position information of an image designated bythe user and that of an image sensed by the same image sensing apparatusas that which sensed this image in place of route information of theimage sensing apparatus. More specifically, an association degreeevaluation unit 204 acquires imaging position information of an imagedesignated by the user from an image metadata management unit 201.Furthermore, the association degree evaluation unit 204 acquires imagingposition information of an image, which is associated with the sameimage sensing apparatus information as that of the image designated bythe user and imaging time information indicating an imaging timeadjacent to that indicated by the imaging time information of this imagefrom the image metadata management unit 201.

Then, the association degree evaluation unit 204 calculates distancesbetween the respective pieces of acquired imaging position informationand respective pieces of position information of a moving object. Inthis calculation, a distance between the imaging position informationand position information at the same time is calculated as in the firstembodiment. Thus, distances at respective times can be calculated.

FIG. 9 is a view for explaining routes of the imaging sensing apparatusand moving objects on a two-dimensional plane. Reference numerals 901 to904 denote positions indicated by pieces of imaging position informationof images associated with pieces of imaging time information indicatingadjacent times of imaging time information of an image designated by theuser. In this case, the association degree evaluation unit 204calculates a distance between a position x (=positions 901 to 904) and aposition on a route 702 at the same time as the position x. Likewise,the association degree evaluation unit 204 calculates a distance betweenthe position x (=positions 901 to 904) and a position on a route 703 atthe same time as the position x. Also, the association degree evaluationunit 204 calculates a distance between the position x (=positions 901 to904) and a position on a route 704 at the same time as the position x.

FIG. 10 shows the distances calculated by the association degreeevaluation unit 204 in association with the routes 702 to 704. Referencenumeral T1 denotes a time indicated by imaging time information at theposition 901; T2, a time indicated by imaging time information at theposition 902; T3, an imaging time of the image designated by the user;T4, a time indicated by imaging time information at the position 903;and T5, a time indicated by imaging time information at the position904. For example, according to FIG. 10, a distance between the position901 and a position on the route 702 at the time T1 as that at theposition 901 is 1.5. Also, a distance between the position 903 and aposition on the route 704 at the time T4 as that at the position 903 is11.3.

Then, similarly to the first embodiment, the association degreeevaluation unit 204 calculates the average values and variance values ofthe distances of the routes 702 to 704, and calculates the degree ofassociation between the image and moving object designated by the userusing these calculated statistical amounts.

Since this embodiment is premised on that a target image is sensed in aseries of operations, images have to be sensed before and after thetarget image (the positions 901 to 904 in FIG. 9). Since the number ofpoints to be compared is more likely to be smaller than that uponcomparison of routes, the calculation volume can be reduced compared tothat upon comparison of routes. Also, since the need for acquiring andsaving routes can be obviated, storage areas and a mechanism for thatpurpose can be reduced. In this embodiment, the number of imaging timesis three. However, the number of imaging times is not particularlylimited as long as it is one or more.

Fourth Embodiment

In the first embodiment, after a degree of association is calculated,the associating unit 205 performs association without intervention ofthe user. However, after the degree of association is calculated, it maybe displayed on a display screen of a display device, and it is inquiredthe user about whether or not to permit association using this degree ofassociation. When the user inputs an instruction to permit association,the processing by the associating unit 205 is then started as in thefirst embodiment.

According to this arrangement, although an additional user operation isrequired compared to the first embodiment, since more intended metadatais easily assigned, there is a merit of enhancing the precision andeffect at the time of re-use.

Fifth Embodiment

In the first embodiment, “information indicating a degree ofassociation” generated by the associating unit 205 is merely registeredin the external storage device 104 as metadata in association with animage. However, this embodiment uses this metadata to correct therecognition result of an object as a moving object. Only differencesfrom the first embodiment will be described below.

FIG. 5 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to thisembodiment. The arrangement shown in FIG. 5 is obtained by adding anobject recognition unit 501 to that shown in FIG. 1. The objectrecognition unit 501 recognizes an object (moving object) using an imageof the object sensed by an image sensing apparatus, and calculates ascore indicating an accuracy of recognition. The accuracy of therecognition result of such object recognition processing varies due tovarious factors such as appearances of the object including distancesfrom the object and orientations of the object, degrees of in-focus, andthe quality and volume of teacher data for recognition.

When the accuracy of the recognition result is insufficient, recognitiondifferent from an actual correct answer is often made. Adopting a wrongrecognition result as the object of the image unwantedly results innoise at the time of re-use, in other words, it causes search errorsupon using object information in, for example, a search. This embodimentuses the degree of association between the object as the moving objectand the image to correct the recognition result of the object.

For example, assume that the recognition results shown in FIG. 11 areobtained by the recognition processing of the object recognition unit501. FIG. 11 shows top three scores according to the accuracies of therecognition results as a result of the recognition processing forobjects by the object recognition unit 501. Assume that the score ofobject X as a recognition result of an object to be recognized exhibitsa largest value in FIG. 11, but object A is a correct answer as arecognition result in practice. At this time, if a degree of associationbetween an image and object A is high upon execution of the processingaccording to the first embodiment to have object A as a moving object,object A and the image are associated with each other since they have astrong association. For example, this score is corrected by multiplyingthe score of object A by a similarity calculated for object A and theimage. The object recognition unit 501 corrects the score.

FIG. 12 shows the correction result of the score of object A shown inFIG. 11 by the object recognition unit 501. As shown in FIG. 12, thescore of object A is corrected to the highest score.

Upon practicing this embodiment, a target moving object of a movingobject route management unit 203 has to be the same as an object to berecognized by the object recognition unit 501. Therefore, therecognition result of the object recognition unit 501 should beassociated with information (object information) unique to the object tobe recognized. Then, whether or not the target moving object of themoving object route management unit 203 is the same as the object to berecognized by the object recognition unit 501 can be judged by checkingwhether or not the object information is the same as the moving objectinformation. Alternatively, for example, an associating unit 205 maymanage a correspondence table used to identify the moving object andobject. In this manner, various methods about a mechanism for specifyingwhether or not the moving object is the same as the object areavailable, and any of these methods may be used.

As described above, according to this embodiment, the quality ofinformation as an object can be improved. Hence, the recognition resultwith insufficient accuracy can be suppressed from being associated withan image as object information. The first embodiment reveals only thepresence of association between the moving object and image. However,according to this embodiment, the recognition result can be associatedas object information. Also, effects infeasible in each of therecognition processing of this embodiment and the associating processingof the first embodiment can be obtained.

Sixth Embodiment

In the above embodiments, still images have been mainly described astargets. However, movies may be used. In case of a movie, processing maybe applied to portions obtained by dividing the entire movie. Forexample, a movie may be divided into scenes, and processing may beapplied to consider each scene as one still image in the aboveembodiments. Alternatively, processing may be applied to every data forn frames. In addition, when latitude information and longitudeinformation are periodically acquired, movie data falling within acertain range may be used as one portion. Of course, one entire moviedata may be handled as a target to be processed.

Seventh Embodiment

This embodiment will explain a display example using an associationbetween an image and moving object described in the first embodiment.Note that only differences from the first embodiment will be describedbelow.

FIG. 14 shows an example in which configurations of subgroups and theirtransitions are extracted from pieces of moving object informationassociated with respective images and they are displayed when a groupincluding a plurality of persons are separated into some subgroups andmake imaging actions. In a practical application case, a plurality ofpersons take a trip or hold an event.

This example shows transitions of subgroups in which six persons A, B,C, D, E, and F started actions, and acted together until the end ofactions. Such display can be used to allow the user to easily input aninstruction to search for an image related to himself or herself or animage to which the user himself or herself was not related (of which heor she did not know). For example, FIG. 14 can be used as a GUI(Graphical User Interface) on which person A clicks a subgroup (forexample, a subgroup 1401) as a combination of persons other than himselfor herself to display images of this subgroup.

FIG. 15 is a block diagram showing an example of the functionalarrangement of an information processing apparatus according to thisembodiment. An image specifying unit 1501 specifies an image group whichbelongs to a specific group from those registered in an external storagedevice 104. For example, the image specifying unit 1501 specifies animage group sensed during, for example, a trip or event. Variousmechanisms for specifying an image group that belongs to a specificgroup are available.

For example, to images to be registered in the external storage device104, labels such as “trip” and “athletic festival” are assigned asmetadata. Thus, when the image specifying unit 1501 specifies imagessensed in, for example, “athletic festival”, it refers to labels ofimages registered in the external storage device 104, and specifiesimages assigned with the label “athletic festival”.

Also, images which were registered at once or at the same date and timein the external storage device 104 may define one group, and a label maybe assigned to each group, as described above. As described in the firstembodiment, when a route of an image sensing apparatus is detected, animage group which belongs to a specific group may be specified based onthe moving range of the image sensing apparatus.

A moving object information management unit 1502 manages pieces ofinformation related to moving objects associated with images. Suchassociation method is not particularly limited. For example, images andmoving objects may be associated by the processing described in thefirst embodiment or may be manually associated by the user. In thisembodiment, assume that the pieces of information related to movingobjects are registered in the external storage device 104 as filesindependently of images.

A moving object association extraction unit 1503 acquires information ofmoving objects associated with each image group specified by the imagespecifying unit 1501 from the moving object information management unit1502. Then, the unit 1503 extracts associations between moving objectsin respective images. In this embodiment, the unit 1503 checks anincrease/decrease in number of moving objects in a series of imagesspecified by the image specifying unit 1501, and extracts information ofmoving objects which were likely to act together and their transitions.

A display unit 1504 generates a screen exemplified in FIG. 14 frominformation of moving objects which are likely to act together and theirtransitions extracted by the moving object association extraction unit1503, and outputs the generated screen to an output device 110 via anoutput device interface 106. FIG. 14 shows a state in which movingobjects which acted together changed according to an increase/decreasein number of moving objects associated with a series of images.

FIG. 16 is a flowchart of processing executed by the informationprocessing apparatus according to this embodiment. In step S1601, theimage specifying unit 1501 specifies an image group which belongs to aspecific group from those registered in the external storage device 104.The user may designate the specific group using an input device 109 orthat specific group may be designated in advance.

In step S1602, the moving object association extraction unit 1503acquires information of a moving object associated with each imageincluded in the image group specified by the image specifying unit 1501from the moving object information management unit 1502.

In step S1603, the moving object association extraction unit 1503extracts information of moving objects associated with respective imagesand their transitions. In this embodiment, the unit 1503 checks anincrease/decrease in number of moving objects associated with a seriesof images specified by the image specifying unit 1501, and extractsinformation of moving objects which were likely to act together andtheir transitions.

In step S1604, the display unit 1504 generates the screen exemplified inFIG. 14 from the information of moving objects which were likely to acttogether and their transitions extracted by the moving objectassociation extraction unit 1503. Then, the display unit 1504 outputsthe generated screen to the output device 110 via the output deviceinterface 106. At this time, a moving object, an action of which cannotbe specified since there is no associated sensed image may appear. Amethod of displaying such moving object is not particularly limited aslong as a message indicating that the action of that moving object isindefinite can be displayed. For example, in FIG. 14, moving objects,actions of which are indefinite, are distinguished by blocks displayedusing broken lines like blocks 1402 (an action of person D isindefinite) and 1403 (an action of person F is indefinite). In addition,blocks may be distinguished by using different colors, by flickering, byusing different fonts, or by displaying messages indicating that anaction is indefinite. Note that the above embodiments may be combined asneeded or may be selectively used as needed.

<Another Method of Calculating Similarity Between Routes of ImageSensing Apparatus and Moving Object>

As described above, other methods of calculating the similarity areavailable, and one of these methods will be described below. Assume thatthe method to be described below collects and manages pieces ofinformation of times and positions of an object (to be referred to asobject position/time information hereinafter) in advance. That is, theconcept “object route” may or may not be used.

A target image is decided, and an image sensing apparatus route adjacentto the target image is acquired, as described above. Next, n points atequal intervals on the image sensing apparatus route are sampled. Then,pieces of object position/time information which are locally andtemporally close to n samples are acquired. Note that “locally close”means that a distance is equal to or smaller than a certain threshold,and “temporally close” means that a time is within a margin defined bycertain thresholds. By the processes executed so far, n sets of piecesof object position/time information can be generated.

When pieces of position/time information of an identical object areincluded in the entire sets obtained in the previous process, it isdetermined that a similarity between the route of that object and theimage sensing apparatus route is high. Note that such judgment may bemade by checking not only whether or not pieces of information areincluded in the n entire sets but also a ratio of the number of piecesof information included in the entire sets. For example when n=10, if80% (=8 sets) of 10 sets include information, it is determined that tworoutes are similar to each other. Also, the aforementioned thresholdsmay be set to be stricter (narrower) with decreasing distance to thetarget.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (for example, computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2009-117046 filed May 13, 2009, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising: afirst acquisition unit which acquires a plurality of pieces of positioninformation of an image sensing apparatus at times adjacent to animaging time of an image by the image sensing apparatus; a secondacquisition unit which acquires, for each of a plurality of movingobjects, a plurality of pieces of position information of the movingobject at the adjacent times; a calculation unit which calculates, foreach of the plurality of moving objects, a similarity between a route ofthe image sensing apparatus and a route of the moving object based onthe plurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object, wherein the similarity indicates a difference between theplurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object, and the similarity is higher as the difference issmaller; and a registration unit which registers, as metadata of theimage, information used to specify the moving object for which thesimilarity is higher than a pre-set threshold even if the moving objectfor which the similarity is higher than the pre-set threshold is notincluded in the image, and wherein the first acquisition unit, thesecond acquisition unit, the calculation unit and the registration unitare implemented by a processor.
 2. The apparatus according to claim 1,further comprising: a first management unit which manages a plurality ofpieces of position information of the image sensing apparatus forrespective times; and a second management unit which manages a pluralityof pieces of position information of the moving object for respectivetimes, wherein said first acquisition unit acquires the plurality ofpieces of position information of the image sensing apparatus at theadjacent times to the imaging time of the image by the image sensingapparatus from the plurality of pieces of position information managedby said first management unit, and said second acquisition unit acquiresthe plurality of pieces of position information of the moving object atthe adjacent times from the plurality of pieces of position informationmanaged by said second management unit.
 3. The apparatus according toclaim 2, wherein said first management unit manages position informationof the image sensing apparatus at the imaging time by the image sensingapparatus.
 4. The apparatus according to claim 2, wherein said firstmanagement unit manages pieces of position information of the imagesensing apparatus, which are measured periodically.
 5. The apparatusaccording to claim 1, wherein said calculation third acquisition unitacquires the similarity using an average value and a variance value ofdistances between the plurality of pieces of position informationacquired by said first acquisition unit and the plurality of pieces ofposition information acquired by said second acquisition unit.
 6. Aninformation processing method implemented by a processor executing: afirst acquisition step of acquiring a plurality of pieces of positioninformation of an image sensing apparatus at times adjacent to animaging time of an image by the image sensing apparatus; a secondacquisition step of acquiring, for each of a plurality of movingobjects, a plurality of pieces of position information of the movingobject at the adjacent times; a calculation step of calculating, foreach of the plurality of moving objects, a similarity between a route ofthe image sensing apparatus and a route of the moving object based onthe plurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object, wherein the similarity indicates a difference between theplurality of pieces of position information of the image sensingapparatus and the plurality of pieces of position information of themoving object, and the similarity is higher as the difference issmaller; and a registration step of registering, as metadata of theimage, information used to specify the moving object for which thesimilarity is higher than a pre-set threshold even if the moving objectfor which the similarity is higher than the pre-set threshold is notincluded in the image.
 7. The apparatus according to claim 1, whereinsaid registration unit registers, in addition to the information used tospecify the moving object for which the similarity is higher than thepre-set threshold, a degree of condition sharing, being based oncloseness of the routes defined by the similarity, between aphotographer of the image and the moving object for which the similarityis higher than the pre-set threshold, as the metadata of the image. 8.The apparatus according to claim 5, wherein said registration unitregisters, as metadata of the image, information used to specify themoving object for which the variance is smaller than a pre-setthreshold.
 9. The apparatus according to claim 1, further comprising: anaccuracy calculation unit which calculates an accuracy when an objectincluded in the image is recognized as the moving object; and acorrection unit which corrects the accuracy based on the similarity. 10.A non-transitory computer-readable storage medium storing a computerprogram for making a computer function as respective units included inan information processing apparatus according to claim 1.